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effort  was  the  implementation  of  a  proof-of-concept  fire  detection  system.  Additional  results 
included  the  development  of  image  processing  modules  capable  of  intensity  and  hue 
thresholding,  low  pass  filtering,  image  subtraction,  region  detection  and  labeling  and  HSI  data 
normalization.  In  the  Phase  I  system,  these  image  processing  modules  were  used  to  filter  and 
format  image  data  for  processing  by  a  neural  network.  Work  conducted  during  the  Phase  I 
program  resulted  in  a  highly  accurate  neural  network  architecture.  The  Phase  I  neurai 
network  was  trained  to  recognize  expanding  fire  regions  within  an  image  using  137  training 
data  sets  consisting  of  96  fire  region  sets  and  41  false  alarm  region  sets.  After  training  was 
completed,  this  network  was  presented  23  test  data  sets  containing  17  fire  regions  and  6  false 
alarm  regions.  The  network  demonstrated  100  percent  accuracy  with  the  training  data  sets 
and  was  also  100  percent  accurate  with  the  test  data  sets.  This  system  was  able  to  demonstrate 
reliable  and  repeatable  detection  of  fire  regions  scaled  to  4  by  4  feet  at  a  range  of  150  feet.  The 
potential  applications  for  this  system,  once  fully  developed  in  Phase  II  of  the  program,  include 
installation  in  facilities  requiring  fire  detection  systems.  For  the  U.  S.  Air  Force,  this  would 
include  aircraft  hangars,  ammunition  depots  and  any  facility  containing  high  value  assets  and 
flammable  materials.  Phase  II  of  the  program  will  accomplish  the  feasibility  established  in 
Phase  I  by  implementing  a  fieldable  fire  detection  system.  Part  of  this  effort  will  include 
converting  the  system  module  algorithms  to  hardware  implementations,  thus  significantly 
increasing  system  processing  speed  and  reducing  fire  detection  times  to  meet  ever  more 
demanding  U.  S.  Air  Force  specifications. 


EXECUTIVE  SUMMARY 


A.  OBJECTIVE 

The  purpose  of  the  work  conducted  under  this  Phase  I  program  has  been  to 
determine  the  feasibility  of  using  machine  vision  techniques  and  neural  network 
computation  to  detect  and  classify  visible  spectrum  signatures  of  fire  hi  the  presence 
of  complex  background  imagery. 

B.  BACKGROUND 

The  U.  S.  Air  Force  has  identified  a  need  for  rapid,  accurate  and  reliable 
detection  and  classification  of  fires.  Standard  fire  detectors  which  rely  on  heat  or 
smoke  sensing  devices  tend  to  be  slow  and  tend  to  react  only  after  the  fire  reaches  a 
significant  size.  Optical  Fire  Detectors  (OFDs)  use  limited-bandwidth  infrared  and 
ultraviolet  electromagnetic  sensing  techniques  to  achieve  the  desired  speed  but  lack 
the  necessary  accuracy.  OFDs  have  repeatedly  caused  the  inappropriate  release  of 
fire  suppression  agents  which  temporarily  render  a  facility  unprotected  and  require 
the  use  of  expensive  replacement  agents.  An  optically-based  fire  detection  systeni 
which  captures  and  intelligently  processes  a  broader  range  of  the  electromagnetic 
spectrum  (such  as  the  visible  region)  would  eliminate  false  alarms  while  retaining 
processing  speeds  comparable  to  those  of  OFD's. 

C.  SCOPE 

The  principal  result  of  Phase  I  was  the  development  of  a  proof-of-concept 
system  which  uses  image  processing  and  neural  network  analysis  techniques  for 
accurate  detection  of  fire  from  complex  video  imagery.  The  system  developed 
consists  of  three  subsystems:  image  capture,  image  processing  and  neural  network 
analysis.  The  image  capture  subsystem  extracts  the  hue,  saturation  and  intensity 
(HSI)  elements  of  a  scene  to  form  a  color  video  image.  The  image  processing 
subsystem  eliminates  all  regions  in  an  image  that  do  not  possess  the  intensity,  hue 
and  growth  characteristics  of  fire.  The  neural  network  analysis  subsystem  compares 
the  hue  and  saturation  patterns  of  remaining  regions  to  the  hue  and  saturation 
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patterns  of  fire  for  a  final  fire/no  fire  determination.  The  proof-of-concept  fire 
detection  system  developed  is  slower  than  conventional  OFD's  since  the  image 
processing  routines  were  implemented  in  computer  software  to  facilitate  algorithm 
development,  testing  and  re-development  for  enhancement  of  detection  accuracy. 

D.  METHODOLOGY 

Color  video  image  capture  systems  have  been  developed  which  extract  either 
MSI  or  red,  green  and  blue  (RGB)  elements  of  a  scene  to  form  a  color  image.  EISI 
format  was  used  in  the  proof-of-concept  system  developed  since  HSI  characteristics 
can  be  correlated  to  fire  imagery  attributes  using  less  complex  processing  algorithms 
than  those  required  for  RGB  format.  Neural  networks  were  used  to  analyze  hue 
and  saturation  patterns  of  video  image  regions  since  the  derivation  of  a 
mathematical  model  to  describe  these  patterns  proved  difficult  and  inaccurate. 

E.  TEST  DESCRIPTION 

The  neural  network  employed  in  the  proof-of-concept  system  developed  was 
trained  to  recognize  the  hue  and  saturation  patterns  of  regions  in  a  video  image  that 
correspond  to  fire  using  137  training  data  sets  consisting  of  96  fire  region  sets  and  41 
false  alarm  region  sets.  After  training  was  completed,  the  network  was  presented  23 
test  data  sets  containing  17  fire  region  sets  and  6  false  alarm  region  sets. 

E.  RESULTS 

The  neural  network  used  in  the  proof-of-concept  system  developed 
demonstrated  100  percent  accuracy  with  the  23  data  sets.  In  addition,  the  system 
demonstrated  accurate  detection  of  fire  regions  scaled  to  a  four-  by  four-foot  area  at  a 
range  of  150  feet. 
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G.  CONCLUSIONS 


Experimental  results  suggest  the  following  conclusions.  The  use  of  HSI  video 
format  as  opposed  to  RGB  video  format  reduces  processing  complexity  and 
simplifies  the  interpretation  of  fire  region  characteristics.  Intensity,  hue  and  growth 
characteristics  of  fire  can  be  used  to  differentiate  regions  in  a  video  image  that 
correspond  to  fire  from  regions  that  correspond  to  most  false  alarms  sources. 
Regions  corresponding  to  fire  can  be  distinguished  from  regions  corresponding  to 
other  false  alarm  sources  through  analysis  of  regional  hue  and  saturation  patterns 
by  a  counter-propagation  uni-flow  neural  network.  Detection  speed  using  the 
algorithms  employed  in  the  Phase  I  proof-of-concept  system  would  be  less  than  five 
seconds  if  the  image  processing  algorithms  are  implemented  in  computer  hardware. 
The  likelihood  of  a  successful  software-to-hardware  conversion  is  high  since  high¬ 
speed  image  processing  hardware  is  readily  available  as  off-the-shelf  components. 
The  cost  of  parts  for  a  hardware-based  Phase  II  prototype  system  would  be  less  than 
$10,000.  The  Phase  III  production  model  system  would  be  of  immediate  use  to  the 
U.S.  Air  Force  for  implementation  in  aircraft  hangars,  ammunition  depots  and 
other  high-risk  installations. 

H.  RECOMMENDATIONS 

The  primary  recommendation  is  conversion  of  the  software-based  Phase  I 
proof-of-concept  system  to  a  hardware-based  Phase  II  prototype  system  to  achieve 
desired  speed  requirements.  Development  of  a  Phase  II  prototype  system  would 
benefit  not  only  the  U.S.  Armed  Forces  but  also  private  industry  as  reliable  fire 
detection  systems  are  needed  to  protect  key  facilities  such  as  hospitals,  where  a  fire 
could  cause  critical  disruption  of  medical  services,  and  nuclear  power  plants,  where 
a  fire  could  result  in  catastrophic  environmental  damage. 
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PREFACE 
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F08635-90-C-0395,  for  the  Air  Force  EngiiVeering  and  Services  Center,  Engineering 
and  Services  Laboratory,  Tyndall  Air  Force  Base,  Florida. 

This  report  summarizes  work  done  between  May  1990  and  November  1990. 
Mr.  Douglas  Schwartz  was  the  AFESC/RDCF  Project  Officer. 
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SECTION  I 


INTRODUCTION 


A.  OBJECTIVE 

The  U.  S.  Air  Force  has  identified  a  need  for  rapid,  accurate  and  reliable 
detection  and  classification  of  fires.  To  address  this  need,  a  proof-of-concept  neural 
network-based,  intelligent  machine  vision  interface  for  the  detection  of  fire 
signatures  in  the  visible  spectrum  was  developed.  The  objective  of  this  Phase  I 
program  has  been  to  determine  the  feasibility  of  using  machine  vision  techniques 
and  neural  network  computation  to  detect  and  classify  visible  spectrum  signatures 
of  fire  in  the  presence  of  complex  background  imagery.  The  benefit  of  this  SBIR 
program  is  the  development  of  a  system  capable  of  real-time  recognition  of  fires 
utilizing  machine  vision  and  neural  network  technology. 

B.  BACKGROUND 

Standard  fire  detectors  which  rely  on  heat  or  smoke  sensing  devices  tend  to  be 
slow  and  to  react  only  after  the  fire  reaches  a  significant  level.  Current 
electromagnetic  sensing  techniques  have  the  desired  speed  but  lack  the  necessary 
accuracy.  Optical  Fire  Detectors  (OFD's)  have  repeatedly  caused  the  inappropriate 
release  of  fire  suppression  agents  which  can  render  a  facility  unprotected  and  require 
the  use  of  expensive  replacement  agents.  In  addition,  OFDs  can  monitor  only  a 
limited  radiation  bandwidth  and  thus  can  collect  only  limited  information  about 
the  presence  of  fire.  The  most  desirable  solution  to  this  problem  takes  advantage  of 
recent  advances  in  machine  vision  and  neural  network  computing  for  application 
in  fire  detection  systems.  The  Phase  I  program  approach  to  these  problems 
employed  machine  vision  techniques  to  generate  digitally  filtered  HSI  (Hue, 
Saturation,  Intensity)  -formatted  video  data.  Once  filtered,  these  data  were  then 
presented  to  an  artificial  neural  network  for  analysis.  Neural  networks  have  been 
used  successfully  in  the  classification  and  recognition  of  complex  input  data  such  as 
those  found  in  pattern  recognition  applications  (Nestor,  1989;  Klimasauskas,  1988; 
Touretzky,  1989). 
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C.  SCOPE 


In  the  Phase  I  program,  positive  results  were  achieved  in  the  application  of 
neural  networks  in  an  intelligent  PISI  video  data  classification  and  analysis  system 
for  the  detection  of  fires.  The  Phase  I  program  demonstrated  the  feasibility  of 
developing  a  neural  network  video  classification  system  which  allows  on-line 
monitoring,  detection  and  classification  of  visible-spectrum  fire  signatures  as 
derived  from  machine  vision  data.  The  principal  result  of  the  Phase  I  effort  was  the 
implementation  of  a  proof-of-concept  fire  detection  system.  Additional  results 
included  the  development  of  image-processing  modules  capable  of  intensity  and 
hue  thresholding,  low-pass  filtering,  image  subtraction,  region  detection  and 
labeling  and  HSI  data  normalization.  In  the  Phase  I  system,  these  image  processing 
modules  were  used  to  filter  and  format  image  data  for  processing  by  a  neural 
network.  A  detailed  description  of  the  system  m.odules  can  be  found  in  later 
sections  of  this  report. 

Work  conducted  during  the  Phase  I  program  resulted  in  a  highly  accurate 
neural  network  architecture.  A  proof-of-concept  neural  network  was  trained  to 
recognize  expanding  fire  regions  within  an  image  using  137  training  data  sets 
consisting  of  96  fire  region  sets  and  41  false  alarm  region  sets.  After  training  was 
completed,  this  network  was  presented  23  test  data  sets  containing  17  fire  regions 
and  6  false  alarm  regions.  The  network  demonstrated  100  percent  accuracy  with 
both  the  training  data  sets  and  the  test  data  sets.  In  addition,  the  system  was  able  to 
demonstrate  reliable  and  repeatable  detection  of  fire  regions  scaled  to  4  by  4  feet  at  a 
range  of  150  feet.  The  potential  applications  for  this  system,  once  fully  developed  in 
Phase  II  of  the  program,  is  the  installation  in  facilities  requiring  fire  detection 
systems.  For  the  U.  S.  Air  Force,  this  would  include  aircraft  hangars,  ammunition 
depots  and  any  facilities  containing  high-value  assets  and  flammable  materials. 
Phase  II  of  the  program  will  accomplish  the  feasibility  demonstrated  in  Phase  I  by 
implementing  a  fieldable  fire  detection  system.  This  effort  will  include  converting 
the  system  module  algorithms  to  hardware  implementations,  thereby  significantly 
increasing  system  processing  speed  and  reducing  fire  detection  times  to  meet  ever 
more  demanding  U.  S.  Air  Force  specifications. 
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The  overall  goal  of  the  Phase  I  research  program  was  to  demonstrate  the 
feasibility  of  applying  neural  network  information  processing  techniques  to 
machine  vision  recognition  of  fire  with  an  accuracy  superior  to  traditional  optical 
monitoring  techniques.  The  innovation  developed  in  this  program  is  the 
application  of  neural  network  technology  as  an  intelligent  interface  to  standard  HSI 
video  data  acquisition  systems  for  the  more  accurate  detection  of  fire  regions  in 
background  light  as  compared  with  currently  available  optical  recognition 
techniques.  The  goal  of  this  research  program  has  been  to  develop  a  sophisticated 
video  data  interface  using  neural  networks  to  provide  simulated  intelligence  in  the 
identification  of  visible-spectrum  fire  signatures.  Specific  objectives  were  as  follows: 

1.  Evaluation  of  the  Machine  Vision  Data 

During  the  completion  of  this  technical  objective,  the  hue,  saturation 
and  intensity  (HSI)  characteristics  of  video  imagery  data  of  fire  regions  and  potential 
false  alarm  regions  were  examined.  Comparison  of  these  regions  with  the  HSI 
attributes  of  the  background  imagery  allowed  the  identification  of  HSI  characteristics 
unique  to  flame-like  regions  within  the  imagery.  Questions  addressed  included 
which  HSI  fire  region  characteristics  were  well-defined  and  therefore  could  be 
processed  by  digital  filtering  and  which  characteristics  needed  the  high  resolution 
analysis  of  the  neural  network. 

2.  Identification  and  Acquisition  of  the  Optimal  Neural  Network  Paradigm 

The  goal  of  this  technical  objective  was  to  determine  and  obtain  the 
optimal  neural  network  paradigm  and  appropriate  simulation  software  for  fire 
detection  applications.  Questions  addressed  included  pattern  recognition, 
associative  memories,  image  processing  and  signal  classification.  All  of  these  issues 
were  considered  in  the  selection  of  an  appropriate  neural  network  architecture. 

3.  Implementation  of  the  Neural  Network  and  Provision  of  User  Interface 

The  network  paradigm  selected  in  Technical  Objective  2  was 
implemented  during  the  completion  of  this  technical  objective.  A 
counter-propagation  paradigm  was  implemented  in  the  Phase  I  system.  Questions 
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cidressed  included  the  number  of  simulated  neurons  per  layer,  the  number  of 
layers,  and  the  number  and  characteristics  of  the  inputs  and  outputs. 

4.  Training  of  the  Selected  Neural  Network  with  the  Video  Flame  Data 

The  neural  network  architecture  implemented  in  Technical  Objective  3 
was  trained  in  this  technical  objective  using  training  data  sets  generated  from  fire 
and  false  alarm  imagery.  The  accuracy  of  the  network  with  various  numbers  of 
inputs  was  observed  and  the  results  of  neural  network  architecture  modifications 
were  compared.  Questions  addressed  during  this  technical  objective  included  the 
size  and  number  of  training  data  sets  and  the  number  of  training  repetitions 
required  before  the  network  accuracy  leveled  off. 

5.  Evaluation  of  Pixel  Geometries  for  Compactness  and  Reliability 

The  input  geometries  and  network  nodes  which  had  no  apparent  or 
deleterious  effects  on  the  accuracy  of  the  network  implemented  in  Technical 
Objective  4  were  removed  and  the  remaining  architecture  was  adjusted  for  optimal 
reliability.  Questions  addressed  during  this  effort  included  the  determination  of 
which  input  and  node  configurations  ensured  maximum  operating  efficiency  and 
speed.  The  identification  of  optical  input  characteristics  was  used  to  define  the 
digital  image  preprocessing  algorithms  required  to  generate  the  desired  input  data 
set  attributes. 

6.  Optimization  of  the  Completed  Neural  Network  System  to  Determine 

Efficiency  and  Validity  of  the  System 

The  completion  of  this  objective  entailed  the  design  and 
implementation  of  a  proof-of-concept,  neural  network-based,  fire  detection  system. 
The  system  was  designed  to  meet  U.S.  Air  Force  time-to-detection  and  resolution 
reejuirements  while  maintaining  significantly  fewer  false  alarm  occurrences,  than 
conventional  optical  fire  detectors  without  degradation  in  fire  detection  sensitivity 
and  accuracy.  Questions  addressed  during  this  technical  objective  included 
characterization  of  realistic  operating  environments  for  which  the  system  is 
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targeted,  image-preprocessir\g  algorithm  speed  and  efficiency  optimization, 
anticipated  false  alarm  scenarios  and  U.  S.  Air  Force  requirements  of  detection  time 
accuracy  and  reliability. 

In  the  Phase  II  program,  the  development  of  an  intelligent  video  data 
classification  and  recognition  interface  will  be  finalized  to  include  a  fieldable  fire 
detection  system  for  use  by  the  United  States  Air  Force.  The  implementation  will 
include  a  hardware-based,  neural  network  system  that  will  accept  video  data  directly 
from  video  capture  hardware  and  yield  an  output  of  the  probability  of  fire  being 
present  and  the  nature  of  the  fire.  This  system  will  be  made  available  commercially 
under  a  Phase  III  program  backed  by  private  or  venture  capital  funding. 
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SECTION  II 


METHODOLOGY 


The  goal  of  Phase  I  of  this  program,  to  evaluate  the  feasibility  of  developing  a 
neural  network  image  analysis  system  for  fire  detection  applications,  involved  a  set 
of  experimental  evaluations  to  satisfy  the  technical  objectives  and  to  establish  the 
basis  for  the  design  of  an  accurate,  reliable  Phase  II  system  to  be  commercialized  in 
Phase  III  of  the  program.  The  Phase  I  technical  program  involved  the  evaluation  of 
I  ISI-formatted  flame  imagery,  the  design  of  a  neural  network  software  architecture 
and  preprocessing  digital  filters  needed  to  implement  the  fire  detection  function,  the 
software  implementation  of  a  proof-of-concept  fire  detection  system  and  the 
evaluation  of  the  Phase  I  prototype  software.  These  topics  are  described  in  the 
following  sections  in  relation  to  the  background  and  associated  work  and  to  the 
work  performed  to  achieve  each  of  the  technical  objectives. 

A.  BACKGROUND  AND  RELATED  WORK 

Standard  color  video  image  processing  usually  is  based  on  techniques  which 
combine  the  red,  green  and  blue  (RGB)  elements  of  color  video  imagery. 
Unfortunately,  RGB  processing  algorithms  represent  an  inherently  complex  and 
computationally  intensive  method  of  processing  digitized  video  data.  More  recently 
color  video  frame  grabbers  have  been  developed  which  are  capable  of  real-time 
video  capture  and  generation  of  digitizpd  imagery  data  in  either  RGB  or  hue, 
saturation  and  intensity  (HSI)  format.  In  RGB  format,  each  pixel  of  a  color  image  is 
characterized  by  three  values  which  correspond  to  the  amount  of  red,  green  and 
blue  needed  to  define  the  color  of  the  pixel.  To  analyze  an  RGB  image,  each  value 
must  be  processed  separately  and  then  recombined  to  characterize  image  attributes. 

The  significance  of  the  HSI  format  is  that  the  hue  data  can  be  correlated  to  the 
visible  spectrum  "color"  of  the  pixel,  while  the  saturation  gives  an  indication  of 
color  "wash  out"  and  the  intensity  indicates  pixel  brightness.  Each  of  these 
characteristics  can  be  independently  correlated  to  fire  imagery  attributes  without 
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complex  processing  algorithms.  This  reduces  the  amount  of  processing  required  to 
detect  fire  regions  in  a  digitized  video  image.  To  capitalize  on  this  reduction  in 
image  processing  complexity  and  the  resulting  increase  in  processing  speed,  the 
Phase  I  fire  detection  system  digital  filtering  modules  process  HSTformatted  video 
imagery.  Once  the  MSI  data  have  been  filtered,  the  imagery  is  processed  by  an 
artificial  neural  network. 

Artificial  neural  neAvorks  are  designed  to  simulate  the  physical  architecture 
of  the  neurons  in  the  human  brain  and  have  demonstrated  significant  performance 
advantages  over  more  conventional  methods  in  pattern  recognition  applications. 
Artificial  neural  networks  differ  from  traditional  pattern  analysis  methodology  in 
that  the  network  can  be  "trained”  by  example  using  correlated  input/output  data 
sets.  The  application  of  neural  network  simulation  technology  to  signal  processing 
is  becoming  standard  practice;  however,  the  widespread  application  of  such 
technology  to  the  field  of  complex  data  interpretation  and  classification  in  video 
image  monitoring  has  not  yet  occurred.  Work  has  been  performed  in  the  area  of 
recognizing  and  classifying  cardiac  sinus  rhythms  (Nestor,  1989;  O'Reilly,  1989)  and 
in  the  area  of  motor  noise  analysis  (O'Reilly,  1989).  However,  these  efforts  have 
centered  on  recognizing  specific  signal  patterns  from  a  single  source  and  not  on  real¬ 
time  monitoring,  classification  and  analysis  of  complex  video  imagery. 

Typical  neural  network  architectures  support  multiple  node  layers  -  an  input 
layer,  an  output  layer,  and  one  or  more  hidden  layers.  The  structure  of  the  hidden 
layers  is  key  to  accurate  and  reliable  network  function.  Too  manv  neurons  in  the 
hidden  layer  will  cause  the  neural  network  simply  to  "memorize"  the  input  data 
files  (Touretzky,  1989;  Obermeier,  1989)  resulting  in  a  netw'ork  which  can  give  the 
proper  response  to  the  training  data  sets,  but  which  cannot  extrapolate  results  for 
data  sets  that  differ  from  the  training  data  sets  (AI  Ware,  1989).  On  the  other  hand, 
too  few  simulated  neurons  in  the  hidden  layers  will  result  in  a  neural  network 
which  will  not  properly  converge  during  training  (Pao,  1989;  Khaidar,  1989; 
Touretzky,  1989;  Obermeier,  1989)  and  which  will  give  proper  results  only  for  the 
most  recent  set  of  training  data,  "forgetting"  training  data  from  the  beginning  of  the 
training  set.  Careful  selection  and  implementation  of  a  network  paradigm  can  lead 
to  a  balance  between  these  two  undesirable  situations. 
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The  ability  to  train  a  neural  network  is  derived  from  multiple 
interconnections  between  weighted  nodes  (neurons).  These  node  weights  are 
randomized  when  the  network  is  initialized  and  are  altered  by  the  network  as 
training  progresses.  This  allows  the  network  to  accord  greater  significance  to  certain 
inputs  and  less  to  others  (Klimasauskas,  1988  and  1989;  Pao,  1989).  In  this  manner  a 
pattern  or  process  is  learned  during  the  training  phase  after  which  the  network  can 
recall  an  associated  output  when  presented  with  an  input  pattern  similar  to  the 
training  data  sets.  When  presented  with  an  input  pattern  that  is  not  an  exact  match 
with  a  pattern  involved  in  the  training,  the  neural  network  is  capable  of  discerning 
which  training  pattern  the  unknown  pattern  m.ost  resembles  and  how  similar  the 
patterns  are.  This  allows  the  neural  network  to  interpolate  input/output 
correlations  as  new  input  patterns  are  encountered. 

Neural  networks  are  computationally  intensive  during  training;  however, 
once  a  network  is  trained,  input  data  can  be  processed  at  extremely  high  throughput 
rates,  making  neural  networks  a  practical  solution  to  real-tin\e  image  processing 
applications.  Neural  network  simulation  is  a  relatively  new  technology  which,  for 
certain  cases,  has  been  able  to  define  relationships  between  data  sets  that  have  defied 
more  conventional  algorithms  and,  in  other  cases,  has  been  able  to  identify 
relationships  which  had  previously  gone  unnoticed.  Herein  lies  the  strength  of 
neural  network  simulations  -  the  ability  to  derive  a  result  for  a  situation  or  pattern 
never  before  encountered  (Klimasauskas,  1988  and  1989). 

The  application  of  neural  network  simulation  technology  to  video  processing 
is  becoming  standard  practice,  but  application  to  the  field  of  complex  data 
interpretation  and  reduction  in  video  monitoring  has  not  yet  been  fully  developed. 
Recent  work  on  complex  signal  analysis  has  been  conducted  by  Anderson  (1990) 
who  used  neural  networks  to  analyze  a  complex  simulated  radar  signal 
environment.  Research  is  also  being  conducted  on  the  use  of  neural  networks  to 
integrate  and  analyze  the  visual  and  acoustic  cues  in  human  speech  for 
machine-based  speech  recognition  applications  (Yuhas,  1990).  Neural 
network-based  acoustic  sensing  has  been  conducted  at  the  Siemens  blower  motor 
manufacturing  facility.  Previous  efforts  to  identify  faulty  motors  relied  on  human 
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monitoring;  a  solution  both  time  consuming  and  labor  intensive.  Conventional 
pattern  recognition  techniques  were  also  inadequate  because  of  insensitivity  to  noise 
differences.  However,  an  artificial  neural  network  system  was  able  to  identify  noise 
problems  in  motors  about  ninety  percent  of  the  time  (O'Reilly,  1989). 

In  a  related  area,  Odin  Corporation  using  neural  networks  was  able  to  identify 
ignition  problems  in  internal  combustion  engines.  Such  problems  were  not 
detected  using  conventional  quality  control  systems  (EET,  1990).  Work  has  also  been 
conducted  in  the  area  of  recognizing  and  classifying  cardiac  sinus  rhythms  (Nestor, 
1989;  O'Reilly,  1989).  These  efforts  centered  on  recognizing  specific  cardiac 
arrhythmias  from  a  single  sensor  as  opposed  to  continuous  monitoring  and 
classification  of  multiple  cardiac  rhythm  signals.  Recent  work  on  superconducting 
neural  networks  has  resulted  in  the  award  of  two  patents  to  the  Naval  Research 
Laboratory  in  Bethesda,  Maryland  (Johnson,  1990). 


B.  EVALUATION  OE  DIGITIZED  VIDEO  IMAGERY 

Technical  Objective  1,  the  evaluation  of  machine  vision  data,  was  approached 
through  an  investigation  of  the  hue,  saturation  and  intensity  (HSI)  characteristics  of 
digitized  imagery  of  flame  regions  and  false  alarm  regions.  This  section  outlines  the 
p’-ogram.  ^hat  was  conducted  to  determine  which  video  data  aspects  arc 
pertinent  to  the  problem  of  visible-spectrum,  machine  vision  fire  detection.  The 
technical  effort  is  divided  into  three  segments:  a)  aspects  of  a  scene  that  should  be 
captured  in  an  image,  b)  aspects  of  an  image  that  should  be  used  to  extract  regions 
corresponding  to  flames  in  the  preprocessing  stage,  and  c)  aspects  of  regions 
corresponding  to  flames  that  should  be  used  as  input  data  to  the  neural  network. 
These  segments  are  discussed  in  the  following  sub-sections. 

1.  Aspects  of  a  Scene  That  Should  Be  Captured  in  an  Image 

Video  acquisition  equipment  can  provide  video  data  in  either  RGB 
(Red,  Green,  Blue)  or  HSI  (Hue,  Saturation,  Intensity)  format.  Hue  is  the  spectral 
aspect  of  a  color  and  ranges  from  red  to  yellow  to  green  to  cyan  to  blue  to  magenta 
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..ad  then  back  to  red.  The  hue  scale  is  a  "circular"  scale  with  red  represented  by  both 
the  lowest  and  highest  limits  of  the  scale.  Saturation  is  the  purity  of  a  color  in 
terms  of  how  faded  or  deep  the  color  appears  to  be.  The  saturation  scale  ranges  from 
completely  faded  (white)  to  completely  pure.  Intensity  is  the  brightness  of  a  color 
and  ranges  from  completely  black  to  completely  while. 

The  HSI  representation  is  considered  superior  to  the  RGB  representation  in 
image  processing  applications  since  the  hue,  saturation  and  intensity  components  of 
a  color  are  highly  uncorrelated,  i.e.,  highly  independent  of  one  another,  whereas  the 
red,  green  and  blue  components  of  a  color  are  highly  correlated.  Because  hue, 
saturation  and  intensity  are  highly  uncorrelated,  information  conveyed  in  the  HSI 
representation  is,  in  many  cases,  easier  to  interpret  than  information  conveyed  in 
the  RGB  representation.  Humans  tend  to  interpret  and  to  describe  color  in  terms  of 
hue  rather  than  in  terms  of  red,  green  and  blue  quantities.  For  example,  the  color 
yellow  is  specified  by  the  term  "yellow"  instead  of  "equal  quantities  of  red  and  green 
with  no  blue."  Since  image  processing  algorithms  are  automated  implementations 
of  how  humans  interpret  images,  more  efficient  algorithms  can  be  developed  as 
image  data  are  better  understood. 

2.  Aspects  of  an  Image  That  Should  Be  Used  to  Extract  Fire  Regions 

Experimentation  has  shown  that  regions  in  color  video  imagery 
corresponding  to  fire  could  be  correctly  extracted  using  hue-  and  intensity-level 
histograms  derived  from  the  image  as  a  whole.  The  hue  levels  displayed  by  regions 
corresponding  to  fire  tend  to  fall  between  red  and  yellow  and  between  magenta  and 
red  as  shown  in  Figure  1.  The  intensity  levels  displayed  by  regions  corresponding  to 
fire  tend  to  be  high  as  shown  in  Figure  2.  The  cutoff  intensity  of  regions 
corresponding  to  fire  is  a  function  of  the  saturation  setting  (not  to  be  confused  with 
the  HSI  saturation  component)  of  the  scanning  camera  employed  and  is 
experimentally  obtained.  Imagery  used  in  the  experimentation  was  obtained  from  a 
high-resolution  color  CCD-based  camera.  The  saturation  levels  of  regions 
corresponding  to  fire  tend  to  occur  at  all  levels  and  thus  can  not  be  used  in  the 
region  extraction  process. 
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gure  2.  Typical  Intensity  Histogram  of  a  Region  Corresponding  to  Fire 


3.  Aspects  of  Fire  Regions  That  Should  Be  Used  as  Input  Data  to  the 

Neural  Network 

Experimentation  has  shown  that  regions  in  a  color  video  image 
corresponding  to  nonfire  phenomena  can  possess  hue  and  intensity  levels  similar 
to  those  possessed  by  regions  corresponding  to  fire.  Phenomena  possessing  hue  and 
intensity  levels  similar  to  those  possessed  by  fire  include  direct  sunlight,  reflection 
of  sunlight  from  glass,  reflection  of  sunlight  from  metal,  reflection  of  sunlight  from 
wood  and  indoor  lights.  Further  experimentation  revealed  that  the  profiles  of  hue- 
and  saturation-level  histograms  derived  from  regions  corresponding  to  fire  tend  to 
differ  in  shape  from  the  curves  of  hue-  and  saturation-level  histograms  derived 
from  regions  corresponding  to  other  light-producing  phenomena.  A  typical 
variation  in  hue  for  regions  of  fire  and  for  false  alarm-regions  is  shown  in  Figure  3, 
while  Figure  4  shows  a  typical  variation  in  saturation.  Curves  of  intensity-level 
histograms  derived  from  regions  corresponding  to  flames  could  not  be 
distinguished  from  the  curves  of  intensity-level  histogram>s  derived  from  regions 
corresponding  to  other  light-producing  phenomena. 

Although  the  difference  in  shape  between  histograms  derived  from  regions 
corresponding  to  two  dissimilar  light-producing  phenomena  may  be  substantial  as 
shown  in  Figure  3,  the  shapes  of  histograms  derived  from  regions  corresponding  to 
similar  light-producing  phenomena  are  not  exact  as  shown  in  Figure  5.  The 
derivation  of  a  mathematical  expression  describing  the  histogram  shape  of  a  region 
corresponding  to  a  particular  light-producing  phenomena  proved  difficult  and,  in 
some  cases,  impossible  for  conventional  image  processing  algorithms.  This  type  of 
complex  data  interpretation  represents  an  application  which  is  well  suited  for 
solution  through  neural  netw’ork  analysis  techniques.  Instead  of  developing  a 
closed  mathematical  description  of  histogram  profiles,  neural  network  analysis 
achieves  a  solution  through  the  construction  of  a  two  dimensional  matrix  of  input 
nodes,  hidden  layers  and  output  nodes.  The  nodes  contained  within  the  matrix  are 
assigned  analysis  "weights"  which  give  certain  analysis  points  or  nodes  more 
influence  on  the  output  value  than  other  nodes.  The  resulting  architecture  can 
thus  develop  a  "best  case"  model  for  the  input/output  correlation  and  provide  a 
probability  assessment  of  the  correct  output  for  a  given  input  set. 
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SELECTION  OF  NEURAL  NETWORK  PARADIGM 


Technical  Objective  2  involved  the  identification  and  acquisition  of  a  neural 
network  paradigm.  Preliminary  efforts  were  directed  toward  the  examination  of 
hue  values  from  video  imagery.  The  hue  histogram  of  an  entire  image  was 
compiled  and  used  as  input  to  a  neural  network.  Hue  histogram  data  were  256  bytes 
long  with  a  single-digit  fire  indicator  as  an  output.  Initial  efforts  employed  a 
network  using  functional-link  expansion  with  back-propagation  of  errors. 

The  functional-link  expansion  has  demonstrated  promise  in  the 
identification  of  complex  patterns  without  the  need  for  excessively  complex 
network  architectures  and  paradigms  (Pao,  1989).  The  functional  link  expansion 
paradigm  is  based  on  an  expansion  of  the  inputs  by  generating  functional 

combinations  of  the  original  input  ••j.ues  and  then  using  these  functional  values  as 
additional  system  inputs,  ihis  process  exponentially  increases  the  number  of 
effective  network  inputs  for  an  cL.phv.'il  inpu*-  data  set.  This  recursive  expansion  of 
the  inputs  has  been  shown  to  eliminate  the  need  for  hidden  network  layers  for 
some  applications  (AI  Ware,  1989). 

During  the  initial  program  efforts,  some  success  was  noted  using  functional 
link  expansion  with  256  hue  histogram  data  inputs;  however,  the  system  was 
limited  due  to  the  inability  of  the  N-Net  210  software  to  utilize  extended  memory 
space  for  the  implementation  of  expanded  network  architectures.  It  was  determined 
that  the  internal  geometries  of  the  pattern  recognition  algorithms  that  needed  to  be 
developed  by  the  neural  network  paradigm  were  too  complex  for  an  unassisted 
linear  network,  such  as  functional  link  expansion,  to  model  adequately.  To  counter 
this  problem,  hidden  layers  were  incorporated  in  the  network  structure.  Hidden 
layers  have  demonstrated  the  ability  to  map  complex  relationships  between  data 
inputs  and  desired  outputs  (Morse,  1989;  Klimasaukas,  1989). 

As  the  network  development  effort  progressed,  it  was  determined  that  a 
significant  increase  in  the  functional  efficiency  of  the  Phase  I  network  could  bo 
achieved  without  significant  degradation  in  analysis  accuracy  by  removing  the 
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functional  link  expansion,  thereby,  leaving  a  counter-propagation  of  errors  model. 
It  was  ascertained  that  the  "function"  of  the  functional  link  expansion  model  could 
be  more  efficiently  duplicated  by  adding  hidden  layers  based  on  the 
counter-propagation  uni-flow  network  paradigm. 

The  structure  of  the  counter-propagation  uni-flow  network  is  shown  in 
Figure  6.  This  type  of  network  maps  a  set  of  normalized  input  vectors,  X,  to  a  set  of 
output  vectors,  Y.  For  the  fire  detection  system  developed,  X  consists  of  Hj^  and 
Hg,  the  normalized  hue  and  saturation  histograms,  respectively,  extracted  from  an 

image  region  obtained  by  the  image  processing  subsystem  that  is  described  in 
SECTION  II,  Subsection  G,  of  this  report.  The  set  of  output  vectors,  Y,  consists  of  the 
vectors  [10  -lO]*^  and  [-10  10]^,  which  correspond  to  the  conditions  "fire"  and  "no  fire", 
respectively.  During  training,  the  Kohonen  layer  measures  the  cosines  of  the  angles 
between  the  input  vectors,  X,  and  a  set  of  weight  vectors,  W.  The  processing 
element  having  the  weight  vector  that  produces  the  smallest  cosine  value  is 
assigned  the  highest  output  value 


F  ire  Fa  Ise  0  larm 


Figure  6.  Counter-Propagation  Uni-Flow  Network  Architecture 
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NEURAL  NETWORK  IMPLEMENTATION 


X 


Technical  Objective  3,  the  implementation  of  the  neural  network  and 
provision  of  a  user  interface,  was  accomplished  with  the  development  of  the  Phase  I 
neural  network  architecture.  The  network  structure  evolved  through  three  distinct 
models.  Initially,  the  neural  iietw-'rk  was  implemented  with  the  AI  Ware  N-Net 
210  system  as  a  two  layer  {no  hid.den  layers)  architecture  with  functional-link 
expansion  of  the  input.  This  network  was  designed  to  accept  256  hue-value  input 
channels  w'hich  were  "functional  link"  expanded  into  over  1200  effective  input 
nodes.  This  model  was  designed  to  detect  fires  and  identify  the  type  of  material 
which  was  burning.  The  output  layer  of  the  network  supported  three  output 
channels,  indicating  the  presence  or  absence  of  burning  wood,  paper,  or  Plexiglass. 
A  high  value  indicated  the  combustion  of  that  particular  fuel,  while  multiple  high 
values  were  manifest  if  there  were  several  fuels  burning.  If  there  were  no  fires,  all 
outputs  would  be  low. 

A  second  version  of  this  architecture  was  developed  when  it  was  determined 
that  tne  network  size  was  larger  than  could  be  supported  by  the  N-Net  210  system. 
The  second  network  model  which  was  deveP  ped  represented  a  modification  of  the 
first  model.  In  this  architecture,  the  number  of  input  channels  was  reduced  to  64 
values  by  averaging  the  original  hue  histogram  values  as  groups  of  four.  The  64 
input  channels  were  "functional  link"  expanded  to  315  input  layer  nodes. 

After  meeting  with  the  Contract  Technical  Monitor  during  the  Three  Month 
Program  Review,  the  definition  of  the  network  analysis  function  was  refined  and 
network  implementation  efforts  converged  on  a  more  specific  application;  namely, 
hangar  fire  detection  with  JP-4  and  JP-5  as  primary  fuel  sources,  ihe  definition  of  a 
more  detailed  system  application  provided  the  basis  for  the  development  of  a  third 
network  architecture  using  a  Neural  Works  Professional  II  development  system. 
The  third  and  final  Phase  I  network  model  is  based  on  counter-propagation  with 
hidden  layers  and  supports  128  input  channels  with  no  functional  link  expansion. 
Sixty-four  of  the  input  channels  provide  hue  histogram  information  while  the 
other  64  provide  saturation  histogram  information.  The  output  layer  supports  two 
output  channels  corresponding  to  a  fire/no  fire  indication. 
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The  finalized  Phase  I  system  architecture  was  designed  to  analyze  possible  fire 
regions  within  an  image.  Region  extraction  is  accomplished  using  digital  filters  for 
preprocessing  analysis  of  the  image  for  the  presence  of  possible  fire  regions  before 
presentation  to  the  network.  These  regions  are  characterized  by  high  intensity, 
unique  hue  and  unique  saturation  spectrums.  Several  sets  of  training  and  test 
imagery  were  generated.  This  imagery  was  selected  to  contain  flame  sources, 
complex  background  structures  and  possible  false  alarm  sources.  The  training  and 
test  imagery  was  captured  using  a  desktop  computer,  commercial  video  digitizing 
hardware,  and  software  developed  for  the  program  effort.  The  system  software, 
which  was  written  in  C,  was  designed  to  capture  an  image,  generate  HSI  histograms 
and  format  the  HSI  data  for  input  to  the  neural  network. 

E.  NEURAL  NETWORK  TIUMNING 

The  completion  of  Technical  Objective  4,  training  of  the  selected  neural 
network  with  the  video  flame  data,  was  accomplished  through  the  training  of  each 
of  the  three  network  models.  The  data  set  for  the  first  network  had  256  hue-value 
input  data  points  and  was  designed  to  detect  various  fuel  sources.  The  input  data 
expansion  to  over  1200  input  nodes  proved  to  be  too  large  for  full  functional-link 
expansion  of  the  net^vork.  The  resulting  limited  expansion  architecture  was  trained 
by  presenting  the  network  with  correlated  input/output  data  sets.  During  the 
training  process,  the  network  self-modified  internal  node  weights  to  correct  output 
errors  using  back-propagation-of-errors  techniques.  The  limited  architecture  of  the 
first  model  was  unable  to  successfully  converge  to  an  acceptable  level  of  accuracy. 
As  training  progressed,  network  accuracy  leveled  off  at  approximately  70  percent  for 
a  fire/no  fire  indication. 

To  address  this  problem,  a  second  network  was  developed  which  represented 
a  modification  of  the  first  network  and  supported  a  reduced  number  (64)  of  input 
channels  with  a  three  channel  output.  The  training  files  produced  for  the  second 
network  had  64  input  values  and  3  corresponding  output  values.  The  64  inputs 
represented  a  four-group  summation  reduction  of  the  256-bin  hue  histogram.  The 
output  fire  indicators  corresponded  to  wood,  paper,  and  Plexiglass  as  fuel  sources. 
Modified  64  element  data  sets  were  used  to  train  this  second  network.  Training 
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^  fforts  on  the  second  network  evaluated  both  back-propagation  of  error  techniques 
and  counter-propagation  of  error  methods.  This  improved  network  architecture 
converged  at  a  more  acceptable  error  level  with  over  90  percent  accuracy  on  fire/no 
fire  decisions  and  75  percent  accuracy  on  fuel  source  classification. 

The  development  of  the  third  and  final  Phase  I  neural  network  model  was 
driven  by  network  optimization  efforts  which  occurred  during  the  completion  of 
Technical  Objective  5.  This  third  model  was  designed  to  address  real-world 
applications  as  outlined  by  the  Contract  Technical  Monitor  such  as  hangar  fire 
detection  in  which  primary  fuel  sources  are  JP-4  and  JP-5  aviation  fuel.  The  final 
Phase  I  network  architecture,  which  does  not  use  functional  link  expansion,  is  based 
on  a  hidden  layer  model  which  supports  counter-propagation  of  errors.  In  view  of 
the  intended  system  application,  training  and  test  imagery  containing  burning  JP-4, 
complex  backgrounds  and  possible  false  alarm  regions  was  digitized.  After  training 
was  completed,  the  final  Phase  I  network  demonstrated  100  percent  accuracy  on  the 
training  and  test  data  sets  which  were  generated  during  Phase  I  of  the  program. 

F.  EVALUATION  OF  NETWORK  ARCHITECTURE,  NODES  AND  INPUT 

DATA  ATTRIBUTES 

In  order  to  accomplish  Technical  Objective  5,  the  evaluation  of  pixel 
geometries  for  compactness  and  reliability,  it  was  recognized  that  input  data 
reduction  was  needed  to  ensure  high  speed  "real-time"  fire  detection  while 
minimizing  false  alarm  incidents.  In  the  Phase  I  system  this  data  reduction  is 
achieved  by  simple  rule  criterion  filtering  to  remove  all  non-flame  like  pixels  before 
image  data  is  presented  to  the  network  for  analysis.  The  "simple  rule"  criteria 
which  were  used  to  reduce  the  input  data  set  defined  possible  "dangerous"  fire 
region  characteristics  as  high  intensity,  unique  hue  spectrum  and  significant  growth. 
If  each  of  these  characteristics  were  present  in  an  image  then  the  image  was  analyzed 
by  the  neural  network  to  ascertain  if  a  region  was  actually  fire  or  was  a  false  alarm. 

Initially,  spatial  analysis  of  imagery  was  based  on  section  by  section  scans  of  a 
rectangular  grid  which  was  overlaid  on  the  image.  Problems  which  were 
encountered  with  this  scanning  technique  included  detection  of  flame  regions 


20 


which  overlap  multiple  segments  and  identification  of  isolated  "safe"  and 
"dangerous"  fire  regions  within  disassociated  segments.  To  address  these  problems, 
a  different  scanning  solution  using  region  ("blob")  analysis  techniques  was 
developed.  Region  analysis  which  uses  no  set  pattern  of  image  division  is  based  on 
the  detection  and  identification  of  associated  regions  in  an  image.  Using  this  type  of 
analysis,  each  object  in  an  image  becomes  an  individual  segment  in  the  scanning 
process.  As  discussed  previously,  "simple  rule"  filtering  is  used  to  reduce  the 
number  of  pixels,  and  thus  image  regions,  which  are  labeled  and  analyzed.  The 
filtering  process  allows  areas  of  interest  in  an  image  to  be  extracted  for  more  detailed 
analysis.  The  application  of  such  data  reduction  techniques  make  possible  the 
development  of  a  faster,  more  efficient  fire  detection  system. 

Once  a  region  is  identified  for  more  detailed  neural  network  analysis,  hue 
and  saturation  characteristics  for  that  region  are  determined.  The  significance  of 
hue  information  in  the  identification  of  fire  regions  was  demonstrated  during  the 
completion  of  earlier  technical  objectives.  To  achieve  even  more  accuracy  in  the 
fire  identification  process,  saturation  information  was  also  provided  the  neural 
network.  The  choice  of  a  combination  of  hue  and  saturation  histogram  information 
as  network  input  data  resulted  in  a  significant  increase  in  network  accuracy.  The 
histograms  were  generated  as  256-item  linear  arrays  which  were  then  reduced  to  64 
items  each  containing  the  sum  of  four  consecutive  array  positions. 

G.  OPTIMIZATION  OF  A  PROOF-OF-CONCEPT  SYSTEM 

Technical  Objective  6,  the  optimization  of  the  neural  network  system  to 
determine  efficiency  and  validity  of  the  system,  was  approached  through  the 
implementation  and  testing  of  the  Phase  I  proof-of-concept  fire  detection  system 
developed  in  the  previous  technical  objectives.  The  region  characteristics  used  to 
identify  possible  fire  regions  within  an  image  before  a  neural  network  analysis  is 
conducted  are:  (1)  a  consistently  high  intensity,  (2)  a  consistently  defined  hue 
bandwidth,  and  (3)  the  tendency  to  expand.  The  flame  characteristic  of  expansion  is 
used  to  distinguish  "safe"  fires  from  dangerous  fires.  Dangerous  fires  tend  to 
expand  in  a  continuous  manner,  whereas  safe  fires,  after  an  initial  expansion  of 
limited  duration,  either  contract  or  remain  constant.  Examples  of  safe  fires  include 


ne  flame  of  a  welding  torch,  the  flame  of  a  cigarette  lighter,  flames  from  the  exhaust 
pipe  of  a  jet,  and  so  on.  A  block  diagram  of  the  proof-of-concept  neural 
network-based  fire  detection  system  developed  in  the  Phase  I  program  is  shown  in 
Figure  7. 


Figure  7.  Phase  I  Proof-of-Concept  Fire  Detection  System 


The  Phase  I  system  fire  detection  function  is  comprised  of  three  subsystems: 
image  capture,  image  processing  and  neural  network  processing.  A  diagram  of  the 
image  capture  subsystem  used  in  the  Phase  1  program  is  shown  in  Figure  8.  Images 
are  digitized  by  a  Pulnix  high  resolution  CCD  camera  with  RGB  output.  The  RGB 
signals  from  the  camera  are  received  by  a  Data  Translation  DT2871  frame  grabber 
board  and  are  converted  into  HSI  format  by  frame  grabber  system  hardware. 
HSI-formatted  images  are  transferred  from  frame  grabber  memory  to  disk  by  a  25 
megahertz  S0386  microprocessor-based  Gateway  2000  desktop  host  computer. 


COLOR  CCD 
CAMER.\ 


Figure  8.  A  Diagram  of  the  Image  Capture  Subsystem 


The  ima^e  processing  subsystem  consists  of  eight  processing  steps.  Beginning 
with  the  capture  of  a  video  frame,  let  C  denote  a  color  digital  image  of  a  scene  that 

may  or  may  not  contain  fire.  Mathematically,  C  is  a  set  of  three  matrices  denoted  by 
■  E^ch  matrix  in  C  is  an  n  by  m  matrix  containing  n  row's  ranging 

from  0  to  n  -1  and  m  columns  ranging  from  0  to  m  -1.  The  parameters  n  and  m  are 
determined  by  the  limitations  of  the  scanning  system  utilized  and  were  480  and  512, 
respectively,  in  the  experiments  performed  during  the  Phase  I  program.  Each 
element  of  ,  C5  and  Cj  (denoted  by  {i,j  ),  C5  (i,; )  and  Cj  {i,j  ),  respectively)  is 
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.in  integer  ranging  from  0  to  2^  -1  where  k  is  an  integer  greater  than  0.  The 
parameter  k  is  the  number  of  bits  used  to  contain  an  image  elenumt  and  is 
determined  by  the  limitations  of  the  scanning  system  utilized.  For  the  Phase  I 
experimentation,  a  k  value  of  eight  (1  byte)  was  used.  The  elements  of  the  matrices 
in  C  ranged  from  0  to  255  and  were  represented  as  unsigned  integer  bytes  during 
computer  processing  and  storage. 

The  element  ii,j  )  of  the  matrix  represents  the  hue  of  the  point  in  the 
scene  scanned  corresponding  to  the  {i,j  )  location  in  C  .  As  explained  in  Technical 
Objective  1,  hue  is  the  spectral  aspect  of  a  color  and  ranges  from  red  to  yellow  to 

green  to  cyan  to  blue  to  magenta  and  then  back  to  red.  The  hue  scale  is  a  circular 
scale  with  red  represented  by  both  0  and  255.  The  element  {i,j  )  of  the  matrix  Cg 

represents  the  saturation  of  the  point  in  the  scene  scanned  corresponding  to  the  {i,j  ) 
location  in  C  .  Saturation  is  the  purity  of  a  color  in  terms  of  how  deep  or  faded  it 

appears  to  be  and  ranges  from  completely  white  (0)  to  completely  pure  (255).  The 
element  Cj  {i,j  )  of  the  matrix  Cj  represents  the  intensity  of  the  point  in  the  scone 

scanned  corresponding  to  the  (i,j  )  location  in  C  .  Intensity  is  brightness  and  ranges 

from  completely  black  (0)  to  completely  white  (255).  Intermediate  levels  of  intensity 
are  gray  level  increments  between  completely  black  and  completely  white  so  that  C/ 

represents  the  black  and  white  version  of  the  color  image  C  .  Figure  9  shows  an 
example  of  a  C/  matrix  that  contains  regions  corresponding  to  flames.  Each  ordered 
triplet  {i,j ),  (i,j  ),  Cj  (i,j  ))=c  (/,/ )  of  C  is  called  a  pixel  of  C. 


The  first  processing  step,  intensity  thresholding,  locates  the  pixels  in  C  that 
have  intensity  levels  equal  to  the  intensity  levels  of  fire.  The  input  of  this 
processing  step  is  Cj .  The  output  is  an  n  by  m  binary  matrix  Tj  where  tj  (i,j  )=I,  if 
cj  {i,j  )  is  greater  than  or  equal  to  while  f/  (i,/  )=0,  if  c/  (i,/  )  is  less  than  ■ 

The  param.eter  lM.in  defined  to  be  the  lowest  intensitv  value  consistently 

observed  for  fire,  and  is  a  function  of  the  camera  saturation  setting.  In  the 
experiments  performed  during  Phase  I,  was  set  to  150.  Figure  10  shows  the  Tj 

matrix  obtained  from  the  image  in  Figure  9  where  gray  represents  the  value  1  and 
white  represents  the  value  0. 


24 


Figure  9.  Intensity  Representation  of  Imagery  Containing  Fire  Regions 


Figure  10.  Intensity  Thresholded  Imagery  of  Fire  Regions 
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The  second  processing  step,  hue  thresholding,  locates  the  pixels  in  C  that 
have  hue  levels  equal  to  the  hue  levels  of  fire.  Experimentation  has  shown  that  the 
hue  levels  of  flames  fall  between  0  (red)  and  50  (yellow),  and  between  200  (magenta) 
and  255  (red).  The  inputs  of  this  processing  step  are  and  Tj .  The  output  is  an 
ri  by  m  binary  matrix  Tj.^ ,  where  ii,j  )=1,  if  Cpj  (/,;  )  is  either  between  0  and  50  or 
between  200  and  255;  otherwise,  hi  h,i  )=^0.  Figure  11  shows  the  matrix  obtained 
from  the  image  in  Figure  9,  where  gray  represents  the  value  1  and  white  represents 
the  value  0. 


Figure  11.  Hue  Thresholded  Imagery  of  Fire  Regions 


The  third  processing  step,  merging  of  thresholded  images,  performs  the 
logical  AN'Ding  of  T^  and  Tpj  .  This  operation  is  illustrated  in  Figure  12.  The 

output  image  of  this  processing  step,  T  ,  is  an  n  by  m  binary  image  where  t  (i ,}  )=1, 
if  tj  {i  ,j  )=1  and  (/ ,/  )=1;  otherwise,  t  (/ )=0.  Thus,  t  (i ,/  )=1  if  the  intensity  value 
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and  the  hue  value  of  the  pixel  at  location  (i  ,j )  in  C  are  within  the  intensity  and  hue 
ranges,  respectively,  defined  for  fire.  Figure  13  shows  the  T  matrix  obtained  from 
the  image  in  Figure  9,  where  gray  represents  the  value  1  and  white  represents  the 
value  0. 
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Figure  12.  The  Logical  ANDing  Operation  of  Tj  and 


Figure  13.  The  T  Matrix  Image  Obtained  from  the  Logical  ANDing  of  Tj  and 
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The  fourth  processing  step,  low-pass  filtering,  removes  high  frequency  values 
from  T,  i.e.  the  edges  in  T  are  smoothed  as  shown  in  Figure  14.  The  purpose  of  this 
processing  step  is  to  remove  high  frequency  noise  caused  by  fire  flicker,  since 
inaccurate  determination  of  fire  expansion  in  the  growth  analysis  processing  step 
has  been  observed  to  occur  if  this  noise  is  unattenuated.  In  addition,  edges  of  fire 
regions  tend  to  be  semi-transparent  and  may  exhibit  corrupted  HSI  characteristics. 
The  output  of  this  processing  step  is  an  n  by  m  binary  matrix  F,  where /(;,;  )=1,  if 
t  a  -1,;  -1)  =  t  a  -1,7  )  =  f  (;■  -1,7  +1)  =  t  ii,j  -1)  =  f  (i,j)  =  t  {i,j  +1)  =  t  (i  +1,7  -1)  =  f  a  +1,7 ) 
=  f  a  +1,/'  +1)=1;  otherwise,  f  (i,j  )=0.  This  operation  is  illustrated  in  Figure  15.  Figure 
16  shows  the  F  matrix  obtained  from  the  image  in  Figure  9,  where  gray  represents 
the  value  1  and  white  represents  the  value  0. 


Figure  14.  Results  of  Low  Pass  Filtering  on  a  Region 
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Figure  15.  Low  Pass  Filtering  Operation 
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BACKGROUND 


Figure  16.  Low  Pass  Filtered  Inaagery  of  Fire  Regions 


The  fifth  processing  step,  image  subtraction,  subtracts  from  F^  _  w'here 
Ff  denotes  the  F  matrix  obtained  from  the  image  scanned  at  time  f .  The  term  Ff 
denotes  the  F  matrix  obtained  from  the  image  scanned  at  time  t  -  At.  Ffere  t 
denotes  time  (in  seconds)  and  denotes  a  predetermined  increment  of  time  (in 
seconds)  that  is  limited  by  the  scanning  speed  of  the  system  utilized.  The 
information  obtained  in  this  processing  step  is  required  for  the  growth  analysis 
operation  performed  in  the  next  processing  step.  The  output  of  this  processing  step 
is  an  n  by  m  integer  matrix  S  where  s  (i )=  (/ ,/  )-/  ^  (i ,/  ).  If  (i  ,j  )=1  and 

/f(i ,/  )^=0,  then  s  (i ,/  )=1  and  flame  contraction  is  implied.  If  /f  (i ,/  )=0  and  (z ,/  ) 
=1,  then  s  (f  )=-l  and  flame  expansion  is  implied.  If  (i  ,j  )=  (i ,;  ),  then  s  (i ,;) 
=0  and  flame  constancy  is  implied.  Figures  17  and  18,  show  black  and  white  images 
scanned  at  times  t  and  t  -  At,  respectively.  Figure  19  shows  the  S  matrix  obtained 
from  the  images  shown  in  Figures  17  and  18  where  black  represents  the  value  1 
(contraction),  gray  represents  the  value  -1  (expansion)  and  white  represents  the 
value  0  (constancy). 
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The  sixth  processing  step,  growth  decision,  determines  whether  there  has 
been  sufficient  flame  growth  in  At  seconds  to  warrant  further  analysis  of  the  image 
scanned  at  time  t  .  The  input  of  this  processing  step  is  S  .  Let  denote  the 

smallest  number  of  8-connected  pixels  that  can  be  correctly  identified  by  the  neural 
network  as  corresponding  to  flames.  The  locations  of  the  pixels  that  are  8-connccted 
to  a  pixel  at  location  (i,/  )  are  defined  to  be  (/  -l,j  -1),  (i  -l,j  ),  (i  -1,;  +1),  (i,;  -1),  (;,/  +1), 
(i  +1,; )  and  (i  +1,/  +1).  If  the  number  of  expansion  elements  in  S  is  greater 
than  or  equal  to  0.05-.\>,  (5  percent  of  ),  the  image  scanned  at  time  t  is  processed 

further;  otherwise,  a  new  image  is  scanned  at  time  t  +Af  and  the  processing  begins 
again  at  the  first  processing  step.  Thus,  the  processing  of  the  image  scanned  at  time  t 
is  continued,  if  and  only  if,  a  fire  growth  size  greater  than  or  equal  to  5  percent  of  the 
area  of  the  smallest  fire  region  detectable  by  the  neural  network  occurs  in  Ar 
seconds. 


Figure  17.  Gray-Scale  Intensity  Imagery  of  Scene  at  Time  "t" 
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Figure  19.  Subtraction  Matrix  of  "t"  fr 


The  seventh  processing  step,  region  detection  and  labeling,  labels  the 
S-connected  regions  believed  to  be  fire  in  the  F  matrix  obtained  from  the  image 
scanned  at  time  t.  The  output  of  this  processing  step  is  an  n  by  m  integer  matrix  R 
where  r  {{,])=  r{k,l)  f  {i,}  )  and  f  (k,l)  belong  to  the  same  8-connected  region; 
otherwise,  r  (i,/ )  does  not  equal  r  {k,l  ).  Region  labeling  is  accomplished  by  first 
setting  the  elements  in  R  equal  to  0.  Let  L  denote  the  current  region  label;  initially, 

L  =0.  Next,  a  row-by-row  and  column-by-column  scan  is  performed  on  F  .  If /(by  )=1 
(possible  flame)  and  r  (i,j  )=0  (unlabeled)  then  L  is  incremented  and  r  (i,j  )  is  set 
equal  to  L  .  The  elements  in  F  that  are  equal  to  1  and  whose  corresponding 
elements  in  R  are  8-connected  to  at  least  one  element  in  R  equal  to  L  are  set  equal 
to  L  .  The  row-by-row,  column-by-column  scan  of  F  is  then  resumed  at  location 
(:',/  -1-1).  After  the  scan  of  F  is  completed,  the  labels  of  8-connected  regions  in  R 
possessing  areas  smaller  than  are  set  equal  to  0  since  fire  regions  possessing 
areas  smaller  that  are  undetectable  by  the  neural  network.  The  Phase  I  neural 
network  was  able  to  detect  fire  area  (Np  )  of  less  than  1,000  pixels  which  scale  to 
approximately  a  four  foot  by  four  foot  fire  region  at  a  range  of  150  feet.  Figure  20 
shows  the  R  matrix  obtained  from  the  image  shown  in  Figure  9  where  white 
represents  the  value  0  and  gray  represents  all  other  values. 

The  eighth  and  final  processing  step,  histogram  extraction,  generates  the  hue 
and  saturation  histograms  from  C  for  each  labeled  region  defined  by  R  .  The  inputs 
of  this  processing  step  are  ,C^  and  R.  Let  H and  Hg  denote  the  hue  and 
saturation  histograms  obtained  from  and  Cg  ,  respectively,  for  a  region  in  R 
defined  by  label  L  .  and  Hg  are  real  vectors  of  size  2^  where  k  is  defined  in 
earlier  paragraphs  in  this  section.  Initially  {i  )  and  /15  (f )  are  set  equal  to  0  for 
I  =0,...,2^  -1.  Then  for  each  r  (/,/  )=L  ,h^i  {a)  and  (b  )  are  incremented  where  a  = 
ci^  (i,i  )  and  b  =  {i,j  ).  The  histograms  are  then  normalized  with  respect  to  area  by 
dividing  each  element  of  H/q  and  by  N ^  where  iV^  is  the  number  of  pixels 
comprising  the  region  in  R  defined  by  label  L  .  Each  sum  (f  )  -f-  /i^  (f  -fl)  -(- 
h^i  a  +2)  a  -t-3)  and  (i  )+h^  (i  -i-l)  +h^{i+2)  +h^  [i  -(-3),  i  =0,4,8,. ..,2^  -1, 

constitutes  an  input  to  the  neural  network  for  the  region  in  R  defined  by  label  L  . 
Adjacent  elements  of  and  Hg  are  combined  in  this  manner  due  to  the  limited 

number  of  neural  network  input  nodes  available  in  practical  networks.  Based  on  the 


information  provided  by  the  image  processing  subsystem,  the  neural  network 
subsystem  makes  a  fire/no  fire  determination  for  the  region  in  R  defined  by  label  L  . 
This  procedure  is  performed  for  all  L  in  R  (except  L  =0). 


BACKGROUND 


Figure  20.  Possible  Fire  Regions  Detected  Within  Imagery 
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SECTION  III 


EXPERIMENTAL  RESULTS 


This  section  contains  a  compilation  of  the  results  of  the  software  and 
hardware  development  and  evaluation  conducted  to  meet  the  Phase  I  Technical 
Objectives.  The  goal  of  this  program  was  the  development  of  a  Phase  I  prototype 
system  which  demonstrates  the  feasibility  of  the  development  of  an  artificially 
intelligent  fire  detection  system  using  neural  network  processing  techniques.  This 
section  describes  the  results  of  the  Phase  I  program  development  effort. 

A.  EVALUATION  OF  DIGITIZED  VIDEO  IMAGERY 

The  completion  of  Technical  Objective  1,  the  evaluation  of  machine  vision 
data,  resulted  in  an  investigation  of  the  hue,  saturation  and  intensity  (HSI) 
characteristics  of  digitized  imagery  of  flame  regions  and  false  alarm  regions.  The 
video  data  studied  consisted  of  the  hue-,  saturation-  and  intensity-level  histograms 
of  images  scanned  under  varying  lighting  conditions.  The  images  which  were 
scanned  contained  regions  corresponding  to  flames  (fueled  by  JP-4,  wood,  paper, 
butane  and  Plexiglass),  direct  sunlight,  reflected  sunlight  from  glass,  reflected 
sunlight  from  metal,  reflected  sunlight  from  wood,  and  indoor  lights.  The  hue 
levels  of  regions  corresponding  to  flames  were  found  to  lie  between  red  (0)  and 
yellow  (50)  and  between  magenta  (200)  and  red  (255).  The  intensity  levels  of  regions 
corresponding  to  flames  were  found  to  lie  above  150  at  maximum  or 
near-maximum  camera  saturation  settings.  The  saturation  levels  of  regions 
corresponding  to  flames  were  found  to  span  the  entire  saturation  scale.  The  curves 
of  hue-  and  saturation-level  histograms  of  regions  corresponding  to  flames  were 
found  to  differ  substantially  in  shape  from  the  curves  of  hue-  and  saturation-level 
histograms  of  regions  corresponding  to  other  light-producing  phenomena.  The 
curves  of  intensity-level  histograms  of  regions  corresponding  to  flames  and  regions 
corresponding  to  other  light-producing  phenomena  could  not  be  distinguished. 


34 


B.  SELECTION  OF  NEURAL  NETWORK  PARADIGM 


The  accomplishinent  of  Technical  Objective  2  involved  the  identification  and 
acquisition  of  a  neural  network  paradigm.  The  completion  of  this  objective  resulted 
in  the  evolutionary  development  of  three  network  models.  The  functional  link 
expansion  paradigm  which  is  based  on  an  expansion  of  the  system  inputs  by 
generating  functional  combinations  of  the  original  input  values  and  then  using 
these  functional  values  as  additional  system  inputs  represents  the  first  network 
paradigm  which  was  developed  during  the  Phase  1  program.  This  model  contained 
no  hidden  layers  and  supported  back-propagation  of  errors  with  over  1200  effective 
input  nodes  and  three  output  nodes.  The  second  model  which  was  developed 
represented  a  modified  version  of  the  first  model  with  128  input  nodes.  The  third 
and  final  network  paradigm  which  was  developed  supported  hidden  layers, 
counter-propagation  of  errors,  128  input  nodes  with  no  functional  link  expansion 
and  a  two  node  fire/no  fire  output  layer. 

C.  NEURAL  NETWORK  IMPLEMENTATION 

The  completion  of  Technical  Objective  3,  the  implementation  of  the  neural 
network  and  provision  of  a  user  interface,  resulted  in  the  development  of  the  Phase 
I  neural  network  architecture.  The  network  structure  evolved  through  three 
distinct  advances  in  paradigm  accuracy,  efficiency  and  speed.  The  initial  network 
model  was  designed  to  support  a  two  layer  (no  hidden  layers)  architecture  with 
functional-link  expansion  of  the  input  corresponding  to  256  hue  value  input 
channels  which  were  expanded  into  over  1200  effective  input  nodes.  This  model 
was  designed  to  detect  fires  and  identify  the  type  of  material  which  was  burning.  It 
was  found  that  the  efficiency  and  speed  of  this  type  of  paradigm  could  be  enhanced 
by  reducing  the  complexity  of  the  input  layer  nodes  from  over  1200  to  64.  This 
reduced  input  paradigm  was  implemented  as  a  second  network  model. 
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Meetings  with  the  Contract  Technical  Monitor  resulted  in  a  more  detailed 
definition  of  the  network  analysis  function  and  network  implementation  efforts 
converged  on  a  more  specific  application;  namely,  hangar  fire  detection  with  JP-4 
and  JP-5  as  primary  fuel  sources.  The  definition  of  a  more  detailed  system 
application  provided  the  basis  for  the  development  of  a  third  netw'ork  architecture 
which  could  analyze  hue  and  saturation  information  and  provide  a  fire/no  fire 
output  response.  During  the  development  of  the  final  Phase  I  network  model, 
training  and  test  data  sets  were  generated  from  video  imagery  containing  regions 
corresponding  to  flames  fueled  by  JP-4,  direct  sunlight,  reflected  sunlight  from  glass, 
reflected  sunlight  from  metal,  reflected  sunlight  from  wood,  and  indoor  lights. 

D.  NEURAL  NETWORK  TRAINING 

The  completion  of  Technical  Objective  4,  training  of  the  selected  neural 
network  with  the  video  flame  data,  resulted  in  the  training  of  each  of  the  three 
network  models.  The  training  data  set  for  the  first  network  had  256  hue  histogram 
bins  representing  imagery  containing  flame  regions  from  various  fuel  sources.  The 
training  process  used  back-propagation  of  error  techniques  to  modify  internal  node 
weights.  This  first  network  model  w'as  unable  to  successfully  converge  to  an 
acceptable  level  of  accuracy.  As  training  progressed,  network  accuracy  leveled  off  at 
approximately  70  percent  for  a  fire/no  fire  indication. 

The  second  network  architecture  represented  a  modification  of  the  first 
network  and  supported  a  reduced  number  (64)  of  input  channels  with  a  three 
channel  output.  The  training  files  produced  for  the  second  network  had  64  hue 
input  values  and  3  corresponding  output  values.  The  64  input  bins  represented  a 
four-group  summation  reduction  of  the  original  256-bin  hue  histogram  input  data 
used  by  the  first  model.  The  output  fire  indicators  corresponded  to  wood,  paper,  and 
Plexiglass  as  fuel  sources.  Training  efforts  on  the  second  network  evaluated  both 
back-propagation  of  error  techniques  and  counter-propagation  of  error  methods. 
This  improved  network  architecture  converged  at  a  more  acceptable  error  level  with 
over  90  percent  accuracy  on  fire/no  fire  decisions  and  75  percent  accuracy  on  fuel 
source  classification. 
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The  third  and  final  Phase  I  network  model  was  designed  to  address  real-world 
applications  as  outlined  by  the  Contract  Technical  Monitor  such  as  hangar  fire 
detection  in  which  primary  fuel  sources  are  JP-4  and  JP-5  aviation  fuel.  The  final 
Phase  I  network  architecture  is  based  on  a  hidden  layer  model  which  supports 
counter-propagation  of  errors.  In  view  of  the  intended  system  application,  training 
and  test  imagery  containing  burning  JP-4,  complex  backgrounds  and  possible  false 
alarm  regions  was  digitized.  After  training  was  completed,  the  final  Phase  I  network 
demonstrated  100  percent  accuracy  for  fire/no  fire  indications. 

E.  EVALUATION  OF  NETWORK  ARCHITECTURE,  NODES  AND  INPUT 

DATA  ATTRIBUTES 

The  completion  of  Technical  Objective  5,  the  evaluation  of  pixel  geometries 
for  compactness  and  reliability,  resulted  in  the  development  of  pre-network 
processing  digital  filtering  algorithms  for  data  reduction  to  ensure  high  speed 
"real-time"  fire  detection  while  minimizing  false  alarm  incidents.  The  result  of  this 
effort  was  the  implementation  of  a  series  of  simple  rule  criterion  filters  to  remove 
all  non-flame  like  pixels  before  image  data  is  presented  to  the  network  for  analysis. 
The  "simple  rule"  criteria  which  were  used  to  reduce  the  input  data  set  defined 
possible  "dangerous"  fire  region  characteristics  as  high  intensity,  unique  hue 
spectrum  and  significant  growth. 

If  each  of  these  characteristics  were  present  in  an  image  then  regions  within 
the  image  were  analyzed  by  the  neural  network  to  ascertain  if  a  region  was  actually 
fire  or  was  a  false  alarm.  This  spatial  scanning  process  which  was  initially  based  on 
uniform  grid  segments  was  later  changed  to  a  different  scanning  solution  using 
region  ("blob")  analysis  techniques.  Problems  which  were  encountered  with  grid 
scanning  included  detection  of  multiple  segment  flame  regions  and  identification  of 
isolated  regions  within  disassociated  segments.  Using  region  analysis,  each 
associated  object  in  an  image  becomes  an  individual  segment  in  the  scanning 
process.  "Simple  rule"  filtering  is  used  to  reduce  the  number  of  pixels,  and  thus 
image  regions,  which  are  labeled  and  analyzed.  The  application  of  the  previously 
discussed  data  reduction  techniques  resulted  in  the  development  of  a  faster,  more 
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eifident  Phase  I  system.  The  final  Phase  I  network  was  trained  to  analyze  both  hue 
and  saturation  characteristics  in  each  region  of  interest.  The  choice  of  a  combination 
of  hue  and  saturation  histogram  information  as  network  input  data  resulted  in  a 
significant  increase  in  network  accuracy. 

F.  OPTIMIZATION  OF  A  PROOF-OF-CONCEPT  SYSTEM 

The  accomplishment  of  Technical  Objective  6,  the  optimization  of  the  neural 
network  system  to  determine  efficiency  and  validity  of  the  system,  resulted  in  the 
implementation  and  testing  of  the  Phase  I  proof-of-concept  fire  detection  system 
developed  in  Technical  Objectives  1  through  5.  Work  conducted  during  the  Phase  I 
program  resulted  in  a  highly  accurate  neural  network  architecture.  The  final  Phase  I 
neural  network  was  trained  to  recognize  expanding  fire  regions  within  an  image 
using  137  training  data  sets  consisting  of  96  fire  region  sets  and  41  false  alarm  region 
sets.  These  data  sets  included  flames  fueled  by  JP-4,  direct  sunlight,  reflected 
sunlight  from  glass,  reflected  sunlight  from  metal,  reflected  sunlight  from  wood, 
and  indoor  lights.  After  training  was  completed,  the  network  was  presented  23  test 
data  sets  containing  17  fire  regions  and  6  false  alarm  regions.  The  network 
demonstrated  100  percent  accuracy  with  the  training  data  sets  and  was  also  100 
percent  accurate  with  the  test  data  sets.  The  system  was  also  able  to  demonstrate 
reliable  and  repeatable  detection  of  fire  regions  scaled  to  4  by  4  feet  at  a  range  of  150 
feet. 
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SECTION  IV 


CONCLUSIONS,  TECHNICAL  FEASIBILITY  AND  RECOMMENDATION 


In  this  section  of  the  report,  major  conclusions  are  presented  based  on  the 
results  of  the  Phase  I  project  and  information  available  during  the  study.  These 
conclusions  are  follo^ved  by  a  discussion  of  the  technical  feasibility  of  each  objective 
and  the  degree  to  which  each  objective  has  been  met.  Finally,  the  potential 
applications  of  the  program  are  discussed  with  regard  to  use  by  the  Federal 
Government  and  in  potential  commercial  applications. 

A.  CONCLUSIONS 

Based  on  the  results  presented  in  this  report  and  other  results  obtained 
during  this  study,  it  is  possible  to  draw  the  following  conclusions. 

•  Experimental  results  suggest  that  HSI  video  format  is  superior  to  RGB  video 
format  in  this  particular  application.  The  use  of  HSI  format  imagery  reduces 
processing  complexity  and  simplifies  the  interpretation  of  region 
characteristics. 

•  Regions  corresponding  to  flames  in  a  video  image  can  be  segmented  from  the 
rest  of  the  image  based  on  a  thresholding  scheme  involving  the  hue-  and 
intensity-level  histograms  of  the  entire  image.  The  use  of  threshold  filtering 
reduces  the  computational  overhead  required  by  video  frame  analysis 
techniques. 

•  The  profiles  of  hue-  and  saturation-level  histograms  of  regions  corresponding 
to  flames  can  differ  significantly  in  shape  from  the  curves  of  hue-  and 
saturation-level  histograms  of  regions  corresponding  to  other  light-producing 
phenomena  while  the  profiles  of  intensity-level  histograms  of  regions 
corresponding  to  flames  and  regions  corresponding  to  other  light-producing 
phenomena  could  not  be  easily  distinguished.  These  findings  indicate  that 
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there  exists  the  possibility  of  using  hue  and  saturation  information  in 
machine  vision  imagery  to  detect  the  presence  of  flame  in  a  complex  false 
alarm  background. 

•  For  a  given  application,  optimization  of  the  neural  network  processing 
architecture  by  restructuring  input  layers,  by  adding  hidden  layers  and  by 
selecting  error  propagation  techniques  can  contribute  significantly  to 
improved  system  performance.  For  hangar  fire  detection  applications,  it  was 
found  that  an  architecture  which  can  process  hue  and  saturation  histogram 
data  using  hidden  layers  and  counter-propagation  of  errors  provides  a  highly 
accurate  fire  detection  capability. 

•  Experimental  results  suggest  that  the  neural  network-based  system  developed 
during  the  Phase  I  program  can  correctly  extract  regions  corresponding  to 
flames  (fueled  by  JP-4)  as  well  as  regions  corresponding  to  other 
light-producing  phenomena  and  can  accurately  identify  each  region  as  a 
"safe"  fire,  a  "dangerous"  fire  or  a  false  alarm  incident. 

•  Experimental  work  conducted  during  the  Phase  I  program  indicates  that  a 
neural  network-based  fire  detection  system  which  uses  machine  vision 
techniques  is  feasible  provided  the  software  algorithms  developed  for  the 
Phase  I  proof-of-concept  system  can  be  successfully  translated  into  high  speed 
hardware.  The  likelihood  of  such  a  software-to-hardware  translation  being 
successful  is  high  as  high  speed  image  processing  hardware  is  readily 
available  as  off-the-shelf  components. 

B.  TECHNICAL  FEASIBILITY  AND  EXTENT  TO  WHICH  PHASE  I  OBJECTIVES 
HAVE  BEEN  MET 

This  discussion  of  technical  feasibility  considers  the  Phase  I  program 
objectives  and  technical  findings  and  the  state  of  the  art  as  reflected  in  the  literature. 

•  Technical  Objective  1  was  achieved  by  evaluating  the  characteristics  of  visible 
spectrum  machine  vision  imagery  of  fire  regions  in  complex  false  alarm 
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backgrounds.  It  was  shown  that  fire  regions  can  be  differentiated  from  false 
alarm  regions  using  the  unique  characteristics  of  region  hue  and  saturation 
histogram  profiles. 

•  Technical  Objective  2  was  achieved  by  the  selection  of  an  optimal  neural 
network  paradigm  for  use  in  the  Phase  I  proof-of-concept  system.  The 
selection  process  began  with  a  functional  link  expansion  architecture  which 
supported  back-propagation  of  errors,  1200  effective  input  nodes  and  no 
hidden  layers.  As  the  program^  progressed,  the  number  of  input  nodes  was 
reduced  and  the  architecture  was  modified  to  support  hidden  layers  and 
counter-propagation  of  errors. 

•  Technical  Objective  3  was  achieved  through  the  implementation  of  the 
neural  network  architectures  which  were  selected  in  the  previous  technical 
objective.  The  development  of  each  of  the  three  architectures  was 
accomplished  successfully  with  full  implementations  of  the  second  and  third 
models  and  a  limited  implementation  of  the  first  model  which  supported 
1200  input  nodes  and  suffered  some  performance  degradation  due  to 
hardware  memory  limitations. 

•  Technical  Objective  4  was  achieved  through  the  training  of  the  neural 
network  architectures  implemented  in  the  previous  technical  objective. 
Training  was  accomplished  using  training  data  sets  generated  from  fire, 
background  and  false  alarm  imagery.  The  accuracy  of  the  network  with 
various  numbers  of  inputs  was  observed  and  the  results  of  neural  network 
architecture  modifications  were  compared. 

•  Technical  Objective  5  was  achieved  with  the  evaluation  and  optimization  of 
various  input  data  geometries.  Input  optimization  efforts  included  the 
development  of  digital  preprocessing  filters  and  the  determination  of  which 
input  and  node  configurations  ensured  maximum  operating  efficiency  and 
speed. 
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•  In  order  to  achieve  Technical  Objective  6,  the  Phase  I  proof-of-concept  neural 
network  system  was  tested  and  refined  for  optimal  performance  in  hangar 
fire  detection  applications.  Optimization  efforts  included  the  improvement 
of  overall  system  miodule  architecture,  enhancement  of  the  preprocessing 
digital  filter  algorithms,  optimization  of  network  input  layer  structure  and 
the  addition  of  hidden  layers  within  the  neural  network  architecture.  Phase  I 
test  results  indicate  that  the  development  of  a  neural  network-based  fire 
detection  system  is  feasible. 

Each  of  the  specified  technical  objectives  for  the  Phase  I  project  was  achieved, 
indicating  the  feasibility  of  the  technical  program  to  develop  a  neural  network-based 
fire  detection  system  using  machine  vision  techniques.  In  conclusion,  the  Phase  I 
program  has  indicated  that  the  development  of  such  a  system  is  feasible  and 
technically  practical. 

C.  RECOMMENDATION 

1.  Potential  Use  by  the  Federal  Government 

The  neural  network  fire  detection  system  being  developed  by  this 
program  can  be  used  by  all  branches  of  the  Federal  Government.  This  system, 
which  is  based  on  a  trainable  neural  network,  is  readily  adaptable  to  numerous 
detection  environments  and  can  find  immediate  use  in  any  application  requiring  a 
fire  detection  system.  It  is  envisioned  that  the  system  would  be  installed  in  high 
risk  areas  where  high  value  assets  and  flammable  materials  present  the  potential  for 
highiv  destructive  fires.  The  completion  of  Phase  II  of  this  program,  which  is 
funded  by  the  U.S.  Air  Force,  would  result  in  a  fieldable  fire  detection  system  capable 
of  immediate  use  in  aircraft  hangars,  ammunition  depots  and  other  high-risk,  high 
value  installations. 

2.  Potential  Commercial  Applications 

Commercial  application  of  the  fire  detection  system  being  developed  by 
this  program  will  probably  occur  initially  in  industry  where  flammable  materials 
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and  ignition  sources  are  often  in  close  proximity.  In  such  cases,  fire  can  represent  a 
significant  threat.  The  system  could  also  be  used  to  protect  "■  /  la  .*nties  such  as 
hospitals,  where  a  fire  could  cause  a  critical  disruption  of  medical  services,  or 
nuclear  power  plants,  where  a  fire  could  result  in  catastrophic  en\'ironmental 
damage.  Since  the  system  is  automated,  applications  migh:  n include  monitoring 
of  remote  installations  where  more  conventional  manual  fire  detection  and 
suppression  methods  are  impractical  or  unavailable.  Other  applications  outside  of 
fire  detection  can  be  found  in  science  where  the  system  can  be  used  for  analysis  of 
chemical  processes  or  in  industry  where  the  system  could  monitor  manufacturing 
processes. 


3.  Follow-On  Funding  Commitment 

A  follow-on  funding  commitment  from  Venture  Resources,  Inc.  for 
Phase  III  commercialization  of  the  neural  network-based  fire  detection  system 
developed  under  Phase  II  of  the  program  has  been  arranged. 
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